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Abstract 



The ranks of most national universities in the annual U.S. News 
“America’s Best Colleges” have not changed significantly during the past 
eight years. This is true even though most national universities have spent 
considerable resources and energy trying to improve their ranks. In this 
paper, we document the stability of the national universities’ ranks since 
1999, describe how a few private universities have been able to improve 
their ranks, and discuss whether or not these techniques can be used by 
other colleges and universities, especially public ones. 



Introduction: 

In 1983, U.S. News published its first rankings of colleges and universities, and except for 1984 
and 1986 has ranked colleges and universities every subsequent year. 1 The 1983 and 1985 
rankings were based on academic reputation alone, but in 1987 U.S. News reduced the weight of 
academic reputation to 25% of its total score, and introduced a series of other measures which it 
combined into a single quantitative formula. The formula worked. U.S. News rankings have 
been enormously successful, so much so that they have spawned a whole cottage industry of 
emulators and competitors - the Washington Monthly rankings, TheCenter rankings from the 
University of Florida, Maclean ’s rankings of Canadian universities, the TimesHigher Education 
Supplement's rankings from London, and from the Institute of Higher Education at Shanghai Jiao 
Tong University in China a list of the top 500 universities in the world. 



1 U.S. News publishes its rankings in late August or early September, but labels them prospectively for the 
following year. This paper follows the U.S. News’ convention and use the ranking year to describe the rankings. 
Thus, 1999 refers to rankings published August 1998; likewise, 2006 refers to rankings that were published August 
2005. To add to the confusion, the data U.S. News collects for the rankings is retrospective. The 2007 rankings, for 
example, will be published August 2006, but they will mostly be based on data from fall 2006 and the prior 2004-05 
(or earlier) academic years. 
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Despite this plethora of imitators and competitors, U.S. News rankings remain the gold standard 
of college and university rankings. Not only were they the first, but the U.S. News rankings have 
garnered attention, credibility, and even legitimacy from college administrators, presidents, and 
trustees across the country who routinely use them as “bragging rights” in their attempts to 
quantify, justify, consolidate, and enhance the quality of their academic programs. They have 
the full attention of every college and university president, including those who claim not to pay 
them much attention. 

U.S. News justifies its rankings in terms of helping prospective students learn more about schools 
they are interested in, and helping them broaden their search to institutions they may never have 
considered, or even heard of. “Combined with college visits, interviews, and your own intuition, 
the rankings can be a powerful tool in the college admissions process.” (U.S. News, Aug. 
29,2005, p.78). However, studies have shown that only the most competitive students pay much 
attention to the rankings when deciding where to apply; most students simply disregard them. 
(McDonough, P., Antionio, A., Walpole, M., & Perez, L., 1997; StudentPoll, 1997). Ironically, 
the greatest impact of the U.S. News rankings may not be upon students (whom they are 
ostensibly designed to serve), but upon college and university administrators. 

The U.S. News annual rankings have also had a significant impact on institutional research 
offices. Institutional researchers at America’s colleges and universities spend seemingly endless 
hours each year analyzing data from U.S. News’ “America’s Best Colleges” issue. The primary 
purpose of all this hard work is to understand what happened to their college’s rank this year and 
to improve their college’s rank next year. Institutional researchers have analyzed, regressed, and 
dissected U.S. News data in an effort to find the “silver bullet” that will instantly propel their 
college to higher and higher ranks. Admittedly, there are some year-to-year fluctuations, but the 
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pattern over the past eight ranking years has been one of great stability, not of great change, in 
the ranks. For the most part, efforts by institutional researchers to improve their institution’s rank 
have been in vain. 

Despite little success, university and college presidents still want to improve their institution’s 
U.S. News rank. Presidents of national universities in the top 20 want to become a “top 10” 
school; those in U.S. News ’ 3 rd tier want to become a 2 nd tier school; and those in the 4 th tier want 
to move up into the 3 tier. Across the country, colleges and universities have adopted 

rd 

formalized, and sometimes informal, goals to rise in the college rankings. The president of 3 
tier Virginia Commonwealth University, for example, was reported to carry a card in his 
briefcase listing his presidential priorities for the next five years. The top one: becoming a tier- 
two university. The trustees had promised him a $25,000 raise if Virginia Commonwealth 
jumped a tier under his watch. At the University of California, Riverside (UCR), the Chancellor 
has placed “To enhance UCR’s reputational rankings” first in her list of campus goals. One of 
the reputational rankings to be enhanced is UCR’s annual ranking in U.S. News’ publication, 
“America’s Best Colleges.” 

As mentioned above, the task of developing a strategy to improve the university’s U.S. News 
rank is often assigned to the campus institutional research office. This is true at UCR, and the 
assignment was very specific — to move UCR from the mid-eighties of U.S. News’ list of national 
universities and into the “Top 50.” To accomplish this task, the Institutional Planning Office 
staff studied the annual ranks of the top 120 national universities between 1999-2006 in order to 
identify institutions that had been able to substantially improve their U.S. News rank, to 

2 As described by Nicholas Thompson, “The best, the top, the most,” New York Times, August 3, 2003, Section 4A, 
pp. 24-27. Virginia Commonwealth University seems not to have succeeded; at least in the 2006 rankings it was still 
ranked in U.S. News’ 3 rd tier for national universities, where it had been at least since 1998. 

3 See < http://www.chancellor.ucr.edu/goals.html >. 
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determine and study the strategies they used, and, if possible, to use those strategies to develop a 
plan for UCR. 

This paper will discuss the steps taken to identify national universities who have improved their 
U.S. News rank, and what they did to improve. Several statistical tests will be described that were 
conducted to test and look for causal relationships between the actions undertaken by universities 
and any change their ranks. Lastly, there is a discussion of the implications of those strategies 
for developing a plan for UCR and other national universities. 

Background: 

The U.S. News ranking scheme is composed of seven major factors - Peer Assessment, 

Retention, Faculty Resources, Student Selectivity, Financial Resources, Graduation Rate 
Performance, and Alumni Giving Rate. Three of those variables - Retention, Faculty Resources, 
and Student Selectivity - are composed of a number of sub-factors. For example, the Retention 
factor includes six-year graduation rates and freshmen retention rates; the Faculty Resources 
factor includes undergraduate class size, faculty compensation, percentage of faculty with Ph.D. 
or other terminal degree, the student/faculty ratio, and the percent of full-time faculty; and the 
Student Selectivity factor includes SAT or ACT scores, high school class standing, and 
acceptance rate. The table below documents the fifteen major variables for national universities 
and national liberal arts as they appeared in the 2006 rankings (published August 29, 2005). 
While variable definitions and weights differed slightly for comprehensive regional institutions, 
the basic structure of the ranking system is the same across all four-year colleges and 
universities. 
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Weight 


Factor 


25% 


Peer Assessment 


20% 


Retention 

- Six-Year Graduation Rate (80%) 

- Freshman Retention Rate (20%) 


20% 


Faculty Resources 

- Class Size Less than 20 (30%) 

- Class Size 50 or Greater ( 10%) 

- Faculty Compensation (35%) 

- Percentage of Faculty with Top Terminal Degree (15%) 

- Student/Faculty Ratio (5%) 

- Percent of Full-Time Faculty (5%) 


15% 


Student Selectivity 

- ACT or SAT Scores (50%) 

- High School Class Standing (40%) 

- Acceptance Rate (10%) 


10% 


Financial Resources 


5% 


Graduation Rate Performance 


5% 


Alumni Giving Rate 



In a paper presented at AIR in 1995, “Changes in College Rankings: How Real Are They,” 
Machung argued that much of the annual changes U.S. News rankings that national universities 
were experiencing at that time were due not to actual changes in their performance, but rather to 
changes in the underlying methodology that U.S. News was using. One of the most notable 
characteristics of national universities, she wrote, is that they do not change that much from year 
to year. Admittedly, they are not static either, but in general, national universities are 
characterized more by continuity than by discontinuity. Yet up through about 1996 the U.S. 
News rankings of many national universities were changing seemingly almost randomly every 
year. 
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Not only did this instability distress many colleges and universities, but it began to undermine 
the credibility and legitimacy of the rankings themselves. Institutional researchers and college 
presidents alike were outraged. In September 1996, in a letter to James Fallows, the newly-hired 
editor of U.S. News, Gerhard Caspar, then president of Stanford, wrote that he was “extremely 
skeptical that the quality of a university can be measured statistically” and argued that such 
movement itself: 

— while perhaps good for generating attention and sales - corrodes the credibility 
of these rankings and your magazine itself. Universities change very slowly, in 
many ways more slowly than even I would like. Yet, the people behind the U.S. 

News rankings lead readers to believe either that university quality pops up and 
down like politicians in polls, or that last year’s rankings were wrong but this 
year’s are right (until, of course, next year’s prove them wrong. 4 

Partly in response to Caspar’s letter, in 1997, U.S. News commissioned the National Opinion 
Research Center (NORC) to undertake a critical review of its methodology. Among other 
issues, NORC recommended that “once U.S. News settles on a methodology, it should remain 
constant unless there is compelling evidence for change.” While NORC did not rule out the need 
or possibility for changes in the future, it did suggest that “the presumption should be against 
change unless there is strong evidence of change in validity of the measures.” 5 

Indeed, U.S. News seems to have listened to NORC and stabilized its methodology. Admittedly, 
there have been a few highly publicized changes since the late 1990s, but these have been 
relatively few and far, especially compared to prior years. However, three significant changes 



4 The letter, initially private, has since entered the public domain and been widely quoted. See 
http://www.topcollegerankings.com/Stanford-usnews.html. 

5 The 1997 NORC report, “A review of the methodology for the U.S. News & World Report’s ranking of 
undergraduate colleges and universities,” was regarded as proprietary information by U.S. News and was not 
released publicly. In the course of researching his 2000 article on the rankings for the Washington Monthly, reporter 
Nicholas Thompson received a copy of the report. The NORC report and U.S. News’ response to it are available 
online at: www.washingtonmonthly.com/features/2000/norc.html. 
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have occurred since 1999. In the 2000 rankings U.S. News standardized all variables in its 
ranking model, a procedure that catapulted CalTech into #1, displacing first-ranked Harvard, 
Princeton and Yale. The following year, after a hefty dose of criticism from baffled readers, 
including some from the displaced Ivies themselves, U.S. News adjusted each school’s research 
spending according to the ratio of its undergraduate to graduate students and applied a 
logarithmic adjustor to deal with “so-called statistical outliers.” CalTech (the statistical outlier) 
was pushed back into fourth, and Harvard, Princeton and Yale were back on top. 6 

The last change occurred in the 2004 rankings when U.S. News eliminated “yield” as a variable 
on its Student Selectivity Index, and distributed its weight among the other three variables on this 
index. U.S. News argued that the weight given to “yield” was so small (1.5% of an institution’s 
total score) that eliminating it would have little effect on the overall rankings. Yield had been a 
controversial variable; critics had charged that schools were adopting early decision programs in 
order to increase their yield (and thus their standing on the U.S. News rankings) to the detriment 
of low-income students, who needed to be able to compare financial aid offers from multiple 
institutions and thus could not commit themselves to one institution in November or December, 
as required by most early decision programs. No longer wanting to be involved in the 
discussions around the proliferation of early decision programs and its effect on low income 
students, U.S. News dropped yield from its ranking model. 

Aside from these three changes - standardization of variables in 2000, adjustments to the 
computation of research expenditures in 2001, and elimination of yield in 2004 - the basic 
structure of the U.S. News ranking model for national universities has been stable since the late 

6 See Amy Graham and Nicholas Thompson, “Broken Ranks,” Washington Monthly, September 2001. Graham and 
Thompson, in fact, predicted in advance that U.S. News might modify its formula in order to drop CalTech back 
below Harvard, Princeton and Yale. “Morse declined to say how the formula has been changed for the rankings that 
will be printed on September 4 th of this year,” he wrote. “But if Caltech's ranking drops and one of the three Ivies 
recovers its crown, read the small print carefully.” 
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1990s. Accordingly, it is hypothesized that most of the changes that have occurred since this 
period of time are due not to changes in U.S. News’ methodology, but rather to actual changes in 
institutional performance, as those are reported to U.S. News. Change, of course, can occur for 
multiple reasons - changes in the methodology used to rank institutions, errors in reporting data, 
or actual changes in institutional performance. However, once the ranking model is stabilized, 
the primary cause of institutional changes in rankings will become change in institutional 
performance on one or more of U.S. News’ indicators. . Since national universities do not change 
very rapidly, it is anticipated that there will be little change in the rankings of national 
universities since the late 1990s. 

It is important to distinguish here between annual changes in the rankings due to short-term 
fluctuations and significant changes over a longer period of time. While schools often do switch 
places on an annual basis, moving up or down a notch or two, most of these annual changes are, 
in fact, “blips” that tend to cancel themselves out over time. Errors in reporting data are a prime 
example of this, since they tend to be found and corrected the following year. Such change is not 
considered to be significant, since an institution’s ranking usually reverts back to its original 
location. 7 Given the stability of national universities, significant change, either upward or 
downward, usually takes multiple years to manifest itself. For purposes of this analysis, 
“significant change” is defined to be consistent change, either up or down, in the U.S. News 
ranking of a national university between 1999 and 2006. The university whose ranks are 
significantly changing will see steady, stable, and long-standing changes in its rank in a 
consistent direction. Small fluctuations of a couple of ranks that do not result in consistent 
movement upward or downward over time are defined as insignificant. Admittedly, nine years is 

7 In the 2006 rankings (published August 2005), for example, UC Davis fell six places, from #42 to #48, due to a 
campus error reporting the percentage of faculty who hold doctorate or professional degrees. The data the campus 
reported to U.S. News indicated that 64% of its faculty held terminal degrees; the correct figure (reported in earlier 
years) was 98%. We expect that in the 2007 rankings, coming out August 2006, that Davis will bounce back to its 
previous position, around #42 or #43. 
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a short period of time in the life of an institution, but it should be long-enough for significant 
institutional change to manifest itself. Moreover, the 1998-2006 ranking years also correspond 
with a period of increased methodological stability in the U.S. News ranking model, making an 
analysis of substantial long-term institutional change (or its absence) in the U.S. News ranking 
model feasible for the first time. 

Methodology 

The data for this study was limited to nine years of data from 1998 through 2006 for national 
universities ranked in the top 50 and to three years of data from 2004 through 2006 for national 
universities ranked between 51 and 120. Values were available for all main categories, i.e. Peer 
Assessment, Graduation Rate Performance, Selectivity Rank, Graduation and Retention Rank, 
Financial Resources Rank, Faculty Resources Rank, and Alumni Giving Rank, but were 
unavailable for several sub-categories including Faculty Compensation, Percentage of Faculty 
with Top Terminal Degree, and Financial Resources. 



Limited availability of data, with much of it in rank format, restricted the types of analyses that 
could be performed and prompted the use of a variety of exploratory analyses. The initial 

o 

analysis involved performing the runs test to demonstrate stability in the ranks . This procedure 
was chosen because it measures longterm stability and was compatible with the data. 

Results from the runs test were then interpreted with a series of exploratory graphs which 
displayed performance measures for universities in the top 50. The approach taken to compute 
the performance measures for this study, mimics U.S. News’ approach for measuring academic 
quality, as based on the input factors of peer assessment, graduation and retention rank, faculty 

8 See, Daniel, Wayne W. (1990). Applied Nonparametric Statistics, Boston: PWS-KENT Publishing Company. 
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resources rank, selectivity rank, financial resources rank, graduation rate performance, and 
alumni giving. This study, however, measured the extent to which universities’ overall academic 
quality has changed over the years instead of examining how universities compare at a specific 
point in time as U.S. News does annually for its rankings publication. 

The decision to mimic U.S. News’ methodology was not made under the assumption that it is 
possible to accurately measure academic quality. Rather, summative measures were needed that 
showed the level of change among universities relative to one another with the computation 
needing to be consistent with the U.S. News ranking scheme. This allowed a connection to be 
established between stability in the rankings and the computed performance measures. In theory, 
U.S. News’ total scores should have been sufficient to provide an explanation but proved 
inadequate because the values were derived from a series of logarithmic and other 
transformations that masked true differences. 

Ideally, the summative measures computed should have employed performance indicator values 
in their simplest form, such as each university’s raw graduation, retention, and acceptance rates 
without having undergone any sort of transformation. Because the complete set of data was 
unavailable, data was used on the seven main performance indicators, including rank data, and 
the corresponding weights U.S. News assigned to them. The performance measures were 
computed using Equation 1. For instance, the 1999 performance measure for the i th university 
was estimated by taking each indicator (Peer Assessment Score, Graduation Rank, Faculty 
Resources Rank, etc.) and computing differences between the 1998 and 1999 values. Once 
differences were calculated for all universities, indicator values for i th university were 
standardized and the performance measure computed. 
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P= 0.25*Peer ( ( yjy .i )i i) - 0.20*Grad(( y>y .i) j i) - 0.20*FacReS(( yjy _i),i) - 0.15*SeleC( (y , y .i ) ,i) 

- 0.10*FinReS(( y , y .i),i) + 0.05*Perform ( ( y;y _ixi) - 0.05*AlGiV(( y)y .p,i) 

for y = 1998, 2000, . . . , 2006 and (1) 

i = 1 to 50 



A scatter plot of performance measures against change in ranks supported the model as a 
reasonable indicator of expected change in rank (See Figure 1) and was more consistent than 
using change in score. The correlation between performance measure and change in rank was 
higher at r=-0.598 than the correlation between change in score and change in rank, r=-0.298 
(See Figure 2). 



Figure 1 Performance Measure and Observed Figure 2 U.S. News Total Score and Corresponding 

Change in Rank for National Universities in top 50. Change in Rank for National Universities in top 50. 
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Following this series of graphical exploratory measures, a procedure called Dominance Analysis 
(DA), developed by Azen and Budescu (2003), was used to supplement the findings and 
determine whether colleges are equally affected by the variables in the U.S. News ranking model. 
For example, DA can be used to determine if Financial and Faculty Resources are as important 
to predicting score for national universities ranked 51 through 120 as they are for national 
universities in the top-50. 
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Using the DA procedure, all possible combinations of pairs of predictor variables were compared 
one at a time. Regression models were run for all possible subsets of the variables, and a 
comparison was made of the contribution to total score made by each predictor in each subset, as 
defined by the squared semi-partial correlation. The predictor variables that contributed the most 
on their own and in the presence of other predictors were defined as the most important. A 
comparison was made of all pairs of predictors until a ranking order of importance was 
established. With each comparison, a bootstrap procedure was used to resample the data and 
compute the probability that one predictor dominated the other. 

There are three levels of dominance - complete, conditional, and general. With complete 
dominance, one predictor contributes more to each model than does the other predictor, 
regardless of model size. Conditional dominance refers to situations where a predictor 
contributes more, on average, to all subset models of similar size. General dominance occurs 
when additional contributions are averaged over all subset models. If the average contribution to 
total score is greater for one predictor then it generally dominates the other. Complete is the 
strongest form of dominance followed by conditional and then general. 

Lastly, a regression procedure was used to develop a model for predicting total score using a 
subset of the indicators; then, the relationship between score and rank was modeled using curve 
estimation. 

Findings 

Stability in the Ranks 

The runs procedure is used to find a series of consistently increasing or decreasing ranks. A 
series is considered random if the ranks vary around a constant mean and there is no pattern of 
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observations trending upwards or downwards. Only the overall ranks for the University of 
Southern California (USC) (z=-2.129, p=0.033) and Washington University in St. Louis (z=- 
2.129, p=0.033) failed the test, indicating their ranks have been steadily increasing while ranks 
for all other top 50 universities have been fluctuating in a manner expected under conditions of 
randomness. 

Exceptions 

USC’s and Washington University’s ability to improve their ranks was partially explained by 
creating a series of performance measures graphs. Figure 3 displays eight-year average 
performance measures plotted against average rank for each national university in the top 50. 
Note that USC is an extreme outlier with an average score of 0.58 and an average rank of 35. It 
can be inferred from the graph that USC’s status as an extreme outlier allowed it to significantly 
improve its rank. Although Washington University’s score is much lower at 0.13, it is also 
somewhat of an outlier given its 8-year average rank of 14. It is important to observe that 
Washington University’s average level of improvement is more than twice that of all other 
national universities in the top 20. 

Yeshiva is also an outlier with an average score of 0.40 and an average rank of 43. Unlike USC 
and Washington University, Yeshiva’s annual performance was inconsistent. It out-performed 
other national universities of similar rank during two years, 1999 and 2002, but wasn’t as 
successful the other six years. Shifts in its yearly rank cancelled each other out. Rensselaer 
Polytechnic also displayed a higher than average score at 0.33 and an average rank of 47. Its 
rank improved by 6 positions between 1999 and 2006. But it failed the runs test due to an 
insufficient number of observations as its rank fell below 50 in 2000. Ranks were not disclosed 
for national universities ranked 51 to 120 at that time. 
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Figure 3 8-year average performance measure by 8-year average rank 
for top 50 national universities, 1999-2006. 



8-Year Average Performance Measure 




Figure 4 illustrates how national universities ranked 1 to 29 have a smaller performance measure 
range than national universities ranked 30 to 50. At rank 30, variability between performance 
measure values increases as overall rank moves toward 50. An investigation of whether annual 
changes in SAT scores, freshmen retention rate, alumni giving rate, etc. also increase as overall 
rank increases was conducted with a series of scatter plots. The graphs showed variability in 
improvement increased with several predictors as well. Further investigation on the relationship 
between individual predictors and sub-ranks showed alumni rank fell nearly exponentially as 
alumni giving rate dropped (R =0.992). Lack of data on individual predictors and unknowns in 
the U.S. News methodology made it impossible to find relationships between other predictors 
and their corresponding sub-ranks. For instance, the fit between graduation and retention rate 
with graduation and retention rank appears to be linear but cannot be confirmed without further 



StabIeRanksPaperV4bl 



Oct. 24, 2006 



15 



knowledge of the U.S. News methodology. The findings do suggest, however, that national 
universities ranked between 1 and 30 have had less relative change in their predictors than 
national universities ranked 30 to 120, however universities ranked 1 to 30 require more relative 
change to see a shift in rank. 

Figure 4 Annual rank by annual performance measure for top 50, 1999-2006. 

Performance Measure by Rank 




■ Top 50 Universities 

♦ use 



A closer examination of USC’s changes over the years reveals it first improved Graduation and 
Retention and Selectivity. As shown in Table lb, substantial changes were observed in 
Acceptance Rate, Freshman in the Top 10% of their High School Class, and SAT Scores. With 
improving graduation rates, USC received one, two, or three extra points from the Graduation 
Rate Performance component for each year since 1999. Alumni Giving increased slightly in the 
beginning, then steadily over the years, so that USC is now ranked 13th best among national 
universities. It appears USC next targeted Faculty Resources, probably by hiring more faculty 
and increasing faculty salaries. This was followed by moving up the Financial Resources Rank. 
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All these changes in rank seemed to affect USC’s academic reputation as well, since its Peer 
Assessment score, which is something not directly under an institution’s control, also rose from 
3.7 (Check this versus Table la) to 3.9.. In sum, and as shown in Table la, USC improved in all 
areas. Washington University’s strategy was similar to USC’s in that improving Student 
Selectivity early was key to its success. In fact, following a successful capital campaign in the 
1990s, Washington University substantially increased its merit aid in order to recruit students 
with higher SAT scores and higher high school GPAs. Not surprisingly, this raised Washington 
University’s rank on U.S. News’ Selectivity measure from 29 to 6. Since students with higher 
SAT scores tend to have higher graduation rates, recruiting better students also helped 
Washington University to improve its graduation and retention rates. This was followed by 
rising on the Financial Resources and Faculty Resources rank. Washington University’s rise on 
the Faculty Resources rank is interesting. The Faculty Resources rank is composed of four 
variables: faculty salaries, undergraduate class sizes, percent full-time faculty, and percent 
faculty with terminal degrees. As shown in Tables 2a and 2b, class size and percent full-time 
faculty did not change over the 1998 to 2006 time period. Since virtually all faculty members at 
national universities have terminal degrees, significant increases in faculty salaries must have 
been responsible for Washington University’s rise in the Faculty Resource Rank from 12 to 5 
between 2002 and 2006. Increasing faculty salaries and allocating more money into scholarships 
and fellowships undoubtedly also helped Washington University rise on U.S. News Financial 
Resources rank as well. In sum, like USC, Washington University improved in nearly all areas. 
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Table la. USC Most Significant Changes Overall 





1998 

Rank 


2006 

Rank 




Overall 

Change 


Alumni Giving Rank 




69 


13 


56 


Selectivity Rank 




70 


19 


51 


Graduation and Retention Rank 




77 


41 


36 


Faculty Resources Rank 




42 


30 


12 


Peer Assessment 




3.8 


3.9 


0.1 


Financial Resources Rank 


(1998-2002) 




37 


49 


-12 


(2002-2006) 




49 


44 


5 



Table 1b. USC Specific Changes 




1998 

Value 


2006 Overall 

Value Change 






1260 


210 


SAT75 


1310 


1440 


130 


Acceptance Rate 


0.72 


0.27 


-0.45 


Freshman Top 10% 


0.43 


0.84 


0.41 


Alumni Giving Rate 


0.20 


0.36 


0.16 


Graduation Rate 


0.69 


0.82 


0.13 


Class Size Less Than 20 


0.50 


0.61 


0.11 


Class Size Greater Than 50 


0.17 


0.11 


-0.06 


Freshman Retention 


0.90 


0.95 


0.05 


Percent Full Time Faculty 


0.82 


0.83 


0.01 


Student Faculty Ratio 


13 


10 


-3 



Table 1c. USC Order of Change 

Selectivity Rank 
Graduation and Retention Rank 

Faculty Resources Rank 
Alumni Giving Rank 

Financial Resouces Rank 

Peer Assessment 



Table 2a. Washington Univ. Most Significant Changes Overall 





1998 


2006 


Overall 






Rank 


Rank 


Change 




Selectivity Rank 




29 


6 


23 


Alumni Giving Rank 




22 


9 


13 


Graduation and Retention Rank 




21 


17 


4 


Financial Resources Rank 










(1998-2000) 




3 


7 


-4 


(2000-2006) 




7 


4 


3 


Faculty Resources Rank 










(1998-2002) 




8 


12 


-4 


(2002-2006) 




12 


5 


7 



Table 2b. Washington University Specific Changes 




1998 

Value 


2006 Overall 

Value Change 




1200 






SAT75 


1400 


1520 


120 


Acceptance Rate 


0.51 


0.22 


-0.29 


Freshman Top 10% 


0.66 


0.93 


0.27 


Graduation Rate 


0.82 


0.92 


0.10 


Alumni Giving Rate 


0.31 


0.39 


0.08 


Freshman Retention 


0.95 


0.97 


0.02 


Student Faculty Ratio 


9 


7 


-2 


Class Size Greater Than 50 


0.08 


0.08 


0 


Class Size Less Than 20 


0.76 


0.74 


-0.02 


Percent Full-Time Faculty 


0.93 


0.92 


-0.01 



Table 2c. Washington 
University Order of Change 



Selectivity Rank 
Alumni Giving Rank 

Financial Resources Rank 

Faculty Resources Rank 

Graduation and Retention Rank 
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Predictors Most Affecting Rank 

While the strategies used by USC and Washington University offer insight into the institutional 
changes required to produce a positive change in rank, predictors which may have been 
important to their strategies may be less important or useful to other national universities, 
particularly those ranked 51 through 120. Dominance Analysis is designed to detect the order of 
predictor importance and it was found the order differed between the top 50 national universities, 
and those ranked greater than 50. (See Table 3). The order differs from the weights assigned by 

U.S. News because DA was able to account for correlation between variables by detecting those 
variables which improve prediction partially due to their correlation with other predictors and 
ranked them accordingly. The main difference among national universities is the high 
importance of Peer Assessment to the top 50 national universities, second only to Selectivity, 
compared to national universities ranked 51 to 120 where Peer Assessment is at the bottom of the 
list and Graduation and Retention Rate is most important. 

Table 3 displays U.S. News weights assigned to the predictors and results from the dominance 
analysis procedure including the average probability a predictor dominates variables of lower 
importance and the confidence level on that probability. The summary is based on the general 
level of dominance. For universities ranked 51 through 120, the results show Graduation and 
Retention Rank conditionally dominates all predictors; and for the top-50 universities, Peer 
Assessment conditionally dominates all but Student Selectivity. 
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Table 3 Predictor importance and the average probability that the predictor dominated each lower ranked variable 
with 1,000 bootstrap samples. 







Ave. Prob. 








Ave. Prob. 








Predictor 








Predictor 








Dominates 








Dominates 






U.S. 


Variables of 




Order of Predictor 


U.S. 


Variables of 




Order of Predictor 


News 


Lower 


Confidence 


Importance Tiers 2 


News 


Lower 


Confidence 


Importance Top 50 


Weight 


Importance 


Level 


and 3 


Weight 


Importance 


Level 










Graduation and 








Student Selectivity 


15% 


0.931 


93% 


Retention Rank 


20% 


0.897 


90% 


Peer Assessment 
Graduation and 


25% 


0.999 


100% 


Student Selectivity 


15% 


0.924 


92% 


Retention 


20% 


0.940 


94% 


Faculty Resources 
Financial 


20% 


0.750 


75% 


Faculty Resources 


20% 


0.953 


95% 


Resources 
Graduation Rate 


10% 


0.801 


80% 


Financial Resources 


10% 


1.00 


100% 


Performance 


5% 


0.799 


80% 


Alumni Giving 
Graduation Rate 


5% 


1.00 


100% 


Peer Assessment 


25% 






Performance 


5% 






Alumni Giving 


5% 







Relationship Between Score and Rank 

Universities that wish to estimate their total score and rank may do so with regression equations. 
But there are three precautions in modeling the equations: 1) the procedure operates under the 
ideal assumptions there will be no changes in the ranking methodology; 2) that other national 
universities’ predictor values have not changed; and 3) that high correlations between variables 
can cause problems if multicollinearity is not controlled. Taking overall score as the dependent 
variable and a subset of indicators as the independent variables, models with R-Squared values in 
the 0.98 to 0.99 range can be achieved with different combinations of predictor variables. The 
predicted score can then be used to estimate the corresponding rank. 

Using 2004 to 2006 data for all national universities, a model was developed to determine the 
relationship between score and rank with curvilinear regression analysis. An exponential model 
was adopted for national universities with U.S. News total scores less than 70 (F=360987), 
p<0.001, R-sq=0.993) and a quadratic curve was adopted for total scores 70 and above (F=4712, 
p<0.001, R-sq=0.992). Figures 5 and 6 display the fit. 
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Figure 5 Exponential fit of scores for 
national universities ranked 30 or higher 



Figure 6 Quadratic fit of scores for 
national universities ranked lower than 30 



U.S. News Rank, 30 to120 



U.S. News, Rank 1 to 29 





The regression models support earlier findings from the performance measures and graphical 
analyses that top ranked universities improve their scores and ranks less easily than do other 
universities. Top ranked universities scores are more stable and a fixed amount of change in 
total score has less effect on their rank than it does for other universities. 

Dominance Analysis further showed predictors affect universities differently despite the fixed 
predictor weights in the U.S. News formula. But it’s almost irrelevant given the findings which 
show that despite continual efforts by universities, ranks have remained stable during the past 
nine years. Given the two exceptions identified, it appears defying the odds requires a long-term 
strategy with a commitment to improve substantially in all areas determined by U.S. News to be 
the indicators of academic quality. 

Discussion: 

The U.S. News rankings have always been stable at the very top. Mel Elfin, the craggy editor 
behind the creation of the U.S. News rankings, used to argue that, of course, the ranking model 
was correct since it confirmed what everybody already knew — that Harvard, Yale and Princeton 
were the best institutions in the country. Change could happen below, but at the very top the 
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waves were always (or almost always) smooth. But since the late 1990s, as has been shown, the 
rankings of the top 120 national universities have also become more stable and more predictable 
as well. Only two institutions — Washington University in St. Louis and the University of 
Southern California in Los Angeles — stand out as exceptions. Since 1998, Washington 
University has risen 6 places and USC 1 1 places. 

How did they do it? Statistical analysis of the data can provide some answers, but anecdotal 
information gleamed from the media also offers insights. Indeed, USC’s and Washington 
University’s rise has not gone unnoticed in the national press. In fact, the Los Angeles Times 
(March 19, 2004) linked USC’s rise from number 42 in 1998 to number 30 in 2004 to a record- 
setting $2.85 billion capital campaign and a “$ 100-million hiring spree” designed, in part, to 
attract top faculty talent. Likewise, the New York Times (Dec. 22, 2003) attributed Washington 
University in St. Louis’s rise from number 20 in 1995 to number 10 in 2003 to a “wildly 
successful” $1.5 billion fund-raising campaign that enabled it to rise on financial resources and 
student selectivity. The New York Times traced Washington University’s rise on student 
selectivity to additional resources it poured into merit-based aid, which enabled the campus to 
recruit a higher caliber student body, thereby raising its SAT scores and graduation rates while 
lowering its admit rate - all significant variables in the U.S. News ranking model. 

But if the analysis presented here is correct, it takes more than a single change on one or two 
variables for a school to rise steadily and consistently in the U.S. News ranking system. As the 
analysis has shown, both Washington University and USC improved on virtually all ranks. The 
stories in the media paint only a partial picture of the underlying structural change these 
institutions seem to have gone through over a period of time. However, they do point out one 
thing both universities had in common: large endowments coupled with a very large and very 
successful capital campaigns — campaigns that enabled them to generate significant amounts of 
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new resources and apply them to specific variables of importance in the U.S. News ranking 
model, especially faculty salaries and merit aid. 9 

Many have noted the resource-driven nature of the U.S. News ranking model: faculty salaries, 
student-faculty ratios, small classes, financial resources, even alumni-giving rates are all a direct 
function of the financial resources available to an institution. So, it should not be surprising to 
find that institutions need substantially more financial resources - above what they already have 
- to really rise in the rankings. But while additional resources may be essential, they are not 
sufficient in themselves, for the funds raised need to be directed into areas where they will 
promote upward movement on U.S. News’ variables. For that, institutions also need financial 
flexibility (often guaranteed by large endowments) as well dedicated, focused, long-term 
commitment coming from the very top. Washington University and USC seem to have had all 
three - the resources, the financial flexibility, and the institutional commitment (coming from 
Chancellor Mark Wrighton and President Steven Sample) to rise in the rankings. 

What about institutional commitment alone? Can it work without significantly large infusions of 
new cash? In two related articles, Ehrenberg (2002, 2000) has described a number of strategies, 
not all of them resource-driven, that Cornell and other national universities and national liberal 
arts colleges have tried in order to raise their U.S. News ranking - from not requiring SAT scores 
for admission (to raise average SAT scores), to encouraging applications from students they are 
not likely to admit (to reduce admit rates), to developing early decision programs (to increase 
yield), to soliciting small donations from many alumni rather than large donations from a few (to 
increase alumni giving rates). While any of these strategies might help a little, none of them are 

9 Eherenberg (2002) argues that faculty salaries, which counts for 35% of USNWR's faculty resources variable, may 
provide part of the reason that faculty salaries in the private research and doctoral universities have risen so 
substantially relative to those in public research universities during the last 20 years: “While salary increases for 
faculty at public research and doctoral universities have been constrained by limitations on state funding, private 
institutions have vigorously increased their spending to enhance their activities and their reputations.” 
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likely to promote significant long-term growth. Why? Because they are relatively easy and 
inexpensive to implement, making it likely that many institutions will implement them, and that 
nobody will gain the necessary competitive advantage. The rapid proliferation of early decision 
programs across private institutions during the 1990s illustrates this point especially well. 10 
Strategies to improve a national university’s rank, then, are a double-edge sword. If all 
institutions apply these strategies, then each school’s rank will not change very much, leading to 
continued stability in the U.S. News ranks. Clearly, the ability of a school to develop a 
successful strategy for rising in the U.S. News rankings is dependent upon a lack of long term 
commitment or the inability of other schools to replicate its success. Hence the importance of 
wildly successful fundraising campaigns. Few institutions can really replicate that. 

The legitimacy of the U.S. News system rests in part upon its stability. While the ranking system 
may have helped some students and parents make more informed decisions about which college 
to attend, it has also (perhaps even more importantly) sharpened minute distinctions in 
institutional prestige and broadcast them widely. Prestige is the coin of the realm in academe, 
leading to the recruitment of better faculty, better students, more research funding, and more 
private funding. Monks and Ehrenberg (1999) found that when a top national private university 
improved in the rankings, it could reduce the amount of institutional grant aid it spent attracting 
its class, but when its rankings fell, it had to spend more to attract its class. In short, changes in 
its U.S. News ranking even affected its financial aid bill. Thus, colleges and universities pay 
inordinate amounts of attention to the rankings. But if this analysis is correct - that it is very 
difficult for national universities to rise significantly in the U.S. News rankings - warning flags 
should be raised about the costs involved in playing this competitive game. 



10 See Machung (1988) for a broader discussion of the issues involved. 
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But costs are involved in not playing the game as well, for schools can not only rise in the 
rankings, they can tumble as well. Maybe their performance on a measure declined, maybe they 
made an error reporting data to U.S. News, or maybe they did nothing to improve. Since the 
rankings are constructed de novo every year, schools are measured not against their performance 
last year, but against how they performed vis-a-vis their peers this year. Thus, an institution’s 
rank can drop even if it submitted exactly the same data it had the previous year. 1 1 Despite the 
large institutional costs involved, it behooves schools to keep improving the data they submit to 
U.S. News - either that, or risk falling behind. In short, given the current stability of the U.S. 
News ranking system and the competitive pressures under which they operate, most national 
universities are running, just to be able to stay in place. 
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